Reading:¶
- (Timeseries Review: PDSW CH3 pp 188-208)
- Dimensionality Reduction: PDSW CH5 In Depth: Principal Component Analysis
- Image Recognition: PDSW CH5 Application: A Face Detection Pipeline
Today¶
- (From last lecture)
- ROC and AUC
- Feature Selection
- Regularization
- Project Reporting
- Time Series
Project Reporting¶
Things to include (recommended)¶
- restate the problem
- get the results upfront
- methodology for reproducibility
- data dictionary
Reporting tools¶
Jupyter notebook (review)¶
- Markup
- Images
- Latex
- Slides
Interactive Graphs¶
- Bokeh
- Plotly
Dashboards¶
- Dash
- Tableau, PowerBI
TimeSeries¶
- Issues
- Datetimes in pandas
- Timestamp
- Timedelta
- Resampling
- Downsampling
- Upsampling
- Shifting
- Windowing
Timeseries Issues¶
- non-iid!
- seasonality
Datetimes in pandas: Timestamp¶
In [2]:
from datetime import datetime
In [3]:
# Year Month Day Hour Minute Second
datetime(2017, 11, 2, 19, 45, 55)
Out[3]:
In [4]:
# pandas can handle many date formats
dates = pd.to_datetime([datetime(2015, 7, 3),
'4th of July, 2015',
'2015-Jul-6',
'07-07-2015',
'20150708'])
dates
Out[4]:
In [5]:
# dates may not get processed on DataFrame creation
df = pd.DataFrame({'date': ['2014-07-04',
'2014-08-04',
'2015-07-04',
'2015-08-04'],
'val': [2,4,6,8]})
df.info()
In [6]:
# can covert using pd.to_datetime()
df.date = pd.to_datetime(df.date)
df.info()
In [7]:
# often useful to set date as index
df.set_index('date', inplace=True)
df
Out[7]:
In [8]:
# can then index by date
df['2014-07-04':'2015-07-04']
Out[8]:
In [9]:
# indexing is also flexible
df['2015']
Out[9]:
Datetimes in pandas: Timedelta¶
In [10]:
# Timedelta: the difference between datetimes
df.index[1] - df.index[0]
Out[10]:
In [11]:
# can operate over entire sets
df.index - df.index[0]
Out[11]:
In [12]:
# works with series as well
s = pd.to_datetime(pd.Series(['2014-07-04','2014-08-04','2015-07-04','2015-08-04']))
s - s[0]
Out[12]:
Example Dataset: Twitter stock price¶
In [13]:
#from pandas_datareader import data
#twtr = data.DataReader('TWTR', start='2013', end='2018',
# data_source='yahoo')
twtr = pd.read_csv('../data/twtr_2013-2018.csv',parse_dates=['Date'])
twtr = twtr.set_index('Date')
twtr.head()
Out[13]:
In [14]:
# for simplicity, just using Close
twtr = twtr.Close
In [26]:
# simple line plot
twtr.plot(figsize=(12,6));
In [27]:
# creating the same plot in plotly using cufflinks
cf.set_config_file(offline=True, world_readable=False, theme='ggplot')
twtr.iplot()
Resampling¶
- downsampling: rougher reslution (days -> months)
- upsampling: finer resolution (months -> days)
In [21]:
# downsampling to business quarter using resample
# resample allows for summarization (like groupby)
twtr_BQ = twtr.resample('BQ')
twtr_BQ
Out[21]:
In [22]:
# note the frequency and the interval
twtr_BQ.mean().head(3)
Out[22]:
In [23]:
# plotting both
fig = plt.figure(figsize=(12,6))
_ = twtr.plot(style='-', label='by D')
_ = twtr_BQ.mean().plot(style='--', label='by BQ')
_ = plt.legend(loc='upper right');
In [78]:
# plotting using plotly
# need to deal with missing values in twtr_BQ
tmp = pd.DataFrame({'twtr':twtr,'twtr_BQ':twtr_BQ.mean()})
tmp.interpolate().iplot()
In [53]:
# what about upsampling?
# need asfreq to convert from ResamplerObject
twtr_H = twtr.resample('H')
twtr_H.asfreq().head(3)
Out[53]:
In [54]:
twtr_H.ffill().head(3)
Out[54]:
In [55]:
twtr.index
Out[55]:
Windowing¶
In [80]:
# create a centered rolling window of one year
rolling = twtr.rolling(365, center=True)
rolling
Out[80]:
In [81]:
# plot mean and std of one year rolling window
data = pd.DataFrame({'input': twtr,
'one-year rolling_mean': rolling.mean(),
'one-year rolling_std': rolling.std()})
ax = data.plot(style=['-', '--', ':'], figsize=(12,6))
ax.lines[0].set_alpha(0.3)
In [82]:
# create another rolling window, without centering
rolling = twtr.rolling(365, center=False)
data = pd.DataFrame({'input': twtr,
'one-year rolling_mean': rolling.mean(),
'one-year rolling_std': rolling.std()})
ax = data.plot(style=['-', '--', ':'], figsize=(12,6))
ax.lines[0].set_alpha(0.3)
Shifting¶
In [56]:
# apply a frequency to the data
print('original freq: ', twtr.index.freq)
twtr.head(3)
Out[56]:
In [57]:
# setting fill for missing values
twtr_D = twtr.asfreq('D', method='ffill')
print('new freq : ',twtr_D.index.freq)
twtr_D.head(3)
Out[57]:
In [58]:
# shift data 1 year (a year later)
twtr_1Y = twtr_D.tshift(-365)
twtr_1Y.head(3)
Out[58]:
In [79]:
# plotting shifted data
fig = plt.figure(figsize=(12,6))
_ = twtr_D.plot(style='-',label='by D')
_ = twtr_1Y.plot(style='--', label='by D -1Y')
_ = plt.legend()
In [60]:
# calculate return on investment;
# (ending price - beginning price) / beginning price
ROI = (twtr_1Y - twtr_D) / twtr_D *100
fig = plt.figure(figsize=(12,6))
_ = ROI.plot(label='ROI')
plt.ylabel('% Return on Investment');
In [61]:
# plot ROI on shared axis with close
with sns.axes_style('white'):
fig,ax1 = plt.subplots(1,1,figsize=(12,6))
_ = ROI.plot(label='ROI',ax=ax1)
_ = ax1.set_ylabel('% Return on Investment');
ax2 = ax1.twinx()
_ = twtr.plot(style='--',color='r',ax=ax2)
_ = ax2.set_ylabel('Close');
Example:¶
In [110]:
# Gather data of bike activity on Fremont Bridge from 2012-2017
# !curl -o ../data/FremontBridge.csv https://data.seattle.gov/api/views/65db-xm6k/rows.csv?accessType=DOWNLOAD
In [116]:
# load the data
data = pd.read_csv('../data/FremontBridge_2015-2017.csv.zip', index_col='Date', parse_dates=True)
data.head(3)
Out[116]:
In [117]:
# shorten column names and create Total column
data.columns = ['West', 'East']
data['Total'] = data.eval('West + East')
In [193]:
# drop any rows with missing values
print(len(data.dropna())/ len(data))
data.dropna(inplace=True)
In [119]:
# plot data from 2015
data['2015'].plot(style=['-', '--', ':'],figsize=(12,6))
plt.ylabel('Hourly Bicycle Count');
In [120]:
# downsample to weekly sum to smooth things out
weekly = data.resample('W').sum()
weekly.plot(style=[':', '--', '-'], figsize=(12,6))
plt.ylabel('Weekly bicycle count');
In [121]:
# resample at daily for a more granular view
# then apply a rolling window of 30 days to smooth things out
daily = data.resample('D').sum()
daily.rolling(30, center=True).mean().plot(style=[':', '--', '-'], figsize=(12,6))
plt.ylabel('mean hourly count');
In [122]:
# a wider window using a gaussian filter smooths more while
# accentuating daily differences
daily.rolling(50, center=True,
win_type='gaussian').sum(std=10).plot(style=[':', '--', '-'],
figsize=(12,6));
In [123]:
# the same plot in plotly
daily.rolling(50, center=True,
win_type='gaussian').sum(std=10).iplot();
In [126]:
#if we want to only look at time of day
data.index.time
Out[126]:
In [127]:
# get mean data by time (hourly)
by_time = data.groupby(data.index.time).mean()
# create xticks every 4 hours
hourly_ticks = 60 * 60 * 4 * np.arange(6) # sec * min * every4hours
by_time.plot(xticks=hourly_ticks, style=[':', '--', '-'], figsize=(12,6));
In [169]:
# can also look at average by day of week
by_weekday = data.groupby(data.index.dayofweek).mean()
In [184]:
# BUG: can't seem to plot with renamed index
day_names = ['Mon', 'Tues', 'Wed', 'Thurs', 'Fri', 'Sat', 'Sun']
fig,ax = plt.subplots(1,1,figsize=(12,6))
by_weekday.plot(style=[':', '--', '-'], ax=ax);
ax.set_xlabel('Day of Week');
In [185]:
# create a weekend mask
weekend = np.where(data.index.weekday < 5, 'Weekday', 'Weekend')
# get hourly mean values split by weekday, weekend
by_time = data.groupby([weekend, data.index.time]).mean()
In [186]:
# plot weekday and weekend hourly summary separately
import matplotlib.pyplot as plt
fig, ax = plt.subplots(1, 2, figsize=(14, 5))
by_time.loc['Weekday'].plot(ax=ax[0], title='Weekdays',
xticks=hourly_ticks, style=[':', '--', '-'])
by_time.loc['Weekend'].plot(ax=ax[1], title='Weekends',
xticks=hourly_ticks, style=[':', '--', '-']);
